Hello, Jolene! Nice to see you again! :)

My name is Olga. I'm happy to reviewing your project today.

The first time I see a mistake, I' will just point it out and let you find it and fix it yourself. In a real job, your boss will do the same, and I'm trying to prepare you to work as an Data Analyst. But if you can't handle this task yet, I will give you a more accurate hint at the next check.

Below you will find my comments - please do not move, modify or delete them .

You can find my comments in green, yellow or red boxes like this:

Reviewer's comment Success. Everything is done succesfully.

Reviewer's comment Remarks. Some recommendations.

Reviewer's comment Needs fixing. The block requires some corrections. Work can't be accepted with the red comments.

You can answer me by using this:

Student answer.

Overallv reviewer's comment Jolene, thank you for sending your project. You did a great job! I am really impressed! I like your conclusuons and graphs. They are very detailed. Your code is clear, thank you for your work! There are a couple of things that need to be done before your project is complete, but they're pretty straightforward. I believe you can easily fix it! Good luck! :)

Student answer. Hi Olga, Thank you for reviewing my project and for your comments. I want to be a good coder and I appreciate your feedback. I have added the graph you requested. It is a good one to inclue for displaying the event funnel! Thank you, Jolene G.

Overallv reviewer's comment. V.2. Hello Jolene! Thank you for correcting your project! Now your project is a true "A". Congratulations! I added comments with some useful information for your case. Hope, it will be usefully. So, your project has been accepted and you can go to the next sprint. Keep up the good work, and good luck! :)

Integrated Project 2

Project Description

You work at a startup that sells food products. You need to investigate user behavior for the company's app.

Tasks

• Study the sales funnel
  • Find out how users reach the purchase stage.
  • How many users actually make it to this stage?
  • How many get stuck at previous stages?
    • Which stages in particular?
• Examine the results of an A/A/B test
  • The designers would like to change the fonts for the entire app, but the managers are afraid the users might find the new design intimidating. They want to make a decision based on the results of an A/A/B test.
• Find out which set of fonts produces better results.
  • The users are split into three groups: two control groups get the old fonts and one test group gets the new ones.

Important things to consider

• Creating two A groups has certain advantages.
  • We can make it a principle that we will only be confident in the accuracy of our testing when the two control groups are similar.
  • If there are significant differences between the A groups, this can help us uncover factors that may be distorting the results.
  • Comparing control groups also tells us how much time and data we'll need when running further tests.
• Datasets being used in this project:
  • You'll be using the same dataset for general analytics and for A/A/B analysis.
    • In real projects, experiments are constantly being conducted.
    • Analysts study the quality of an app using general data, without paying attention to whether users are participating in experiments.

Description of the data

• File path: /datasets/logs_exp_us.csv
• Each log entry is a user action or an event.
  • EventName — event name
  • DeviceIDHash — unique user identifier
  • EventTimestamp — event time
  • ExpId — experiment number: 246 and 247 are the control groups, 248 is the test group

Instructions for completing the project

Step 1. Open the data file and read the general information

• Step 1. Conclusion

Step 2. Prepare the data for analysis

• 2.a Rename the columns in a way that's convenient for you
• 2.b Check for missing values and data types. Correct the data if needed
• 2.c Check for and process duplicate rows
  • 2.c Conclusion
• 2.d Add a date and time column and a separate column for dates
• Step 2. Conclusion

Step 3. Study and check the data

• 3.a How many events are in the logs?
• 3.b How many users are in the logs?
• 3.c What's the average number of events per user?
• 3.d What period of time does the data cover?
  • 3.d.1 Find the maximum and the minimum date.
  • 3.d.2 Plot a histogram by date and time.
  • 3.d.3 Find the moment at which the data starts to be complete and ignore the earlier section.
    • Can you be sure that you have equally complete data for the entire period?
    • Older events could end up in some users' logs for technical reasons, and this could skew the overall picture.
  • 3.d.4 What period does the data actually represent?
• - 3.e Did you lose many events and users when excluding the older data?
  • 3.e Conclusion
• 3.f Make sure you have users from all three experimental groups.
  • 3.f Conclusion
• Step 3. Conclusion

Step 4. Study the event funnel

• 4.a See what events are in the logs and their frequency of occurrence. Sort them by frequency.
  • 4.a Conclusion
• 4.b Find the number of users who performed each of these actions. Sort the events by the number of users.
  • 4.b Conclusion
  • 4.b.1 Calculate the proportion of users who performed the action at least once.
    • 4.b.1 Conclusion
• 4.c In what order do you think the actions took place. Are all of them part of a single sequence? You don't need to take them into account when calculating the funnel.
  • 4.c Conclusion
• 4.d Use the event funnel to find the share of users that proceed from each stage to the next. (For instance, for the sequence of events A → B → C, calculate the ratio of users at stage B to the number of users at stage A and the ratio of users at stage C to the number at stage B.)
  • 4.d Conclusion
• 4.e At what stage do you lose the most users?
  • 4.e Conclusion
• 4.f What share of users make the entire journey from their first event to payment?
  • 4.f Conclusion
• Step 4. Conclusion

Step 5. Study the results of the experiment

• 5.a How many users are there in each group?
• 5.b Determine if there is a statistically significant difference between sample groups
  • 5.b.1 Create a function to find the following for each action in the funnel
    • find the number of users who performed the action
    • find the share of users who performed the action
    • test if the groups are statistically significant different
  • 5.b.2 Use the function to do:
    • A/A test
      • A/A test conclusion
    • A1/B test
      • A1/B test conclusion
    • A2/B test
      • A2/B test conclusion
    • A combo/B test
      • A combo/B conclusion
  • 5.b Conclusion
• 5.c What significance level have you set to test the statistical hypotheses mentioned above?
  • 5.c.1 Calculate how many statistical hypothesis tests you carried out.
  • 5.c.2 With a statistical significance level of 0.1, one in 10 results could be false. What should the significance level be?
    • 5.c.2 Conclusion
  • 5.c.3 If you want to change it, run through the previous steps again and check your conclusions.
    • 5.c.3 Conclusion
• Step 5. Conclusion

Step 6. Make a general conclusion

Reviewer's comment Great start with detailed introduction!

Import Libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import datetime
import seaborn as sns
from plotly import graph_objects as go
import scipy.stats as st
import math as mth

import sys
import warnings
if not sys.warnoptions:
    warnings.simplefilter("ignore")
    
sns.set_palette('bright')
pd.set_option('max_colwidth', 450)

Reviewer's comment Thank you for install and import libraries at the beginning.

Step 1. Open the data file and read the general information

Plan of Action

• open and save original data as a dataframe
• examine general info
• take a closer look at the individual columns

Open and save original data as a dataframe

# read and open original data as df
events_data_orig = pd.read_csv('datasets/logs_exp_us.csv', delimiter = "\t")

Examine general info

# review first few rows
events_data_orig.head(10)

	EventName	DeviceIDHash	EventTimestamp	ExpId
0	MainScreenAppear	4575588528974610257	1564029816	246
1	MainScreenAppear	7416695313311560658	1564053102	246
2	PaymentScreenSuccessful	3518123091307005509	1564054127	248
3	CartScreenAppear	3518123091307005509	1564054127	248
4	PaymentScreenSuccessful	6217807653094995999	1564055322	248
5	CartScreenAppear	6217807653094995999	1564055323	248
6	OffersScreenAppear	8351860793733343758	1564066242	246
7	MainScreenAppear	5682100281902512875	1564085677	246
8	MainScreenAppear	1850981295691852772	1564086702	247
9	MainScreenAppear	5407636962369102641	1564112112	246

events_data_orig.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 244126 entries, 0 to 244125
Data columns (total 4 columns):
 #   Column          Non-Null Count   Dtype 
---  ------          --------------   ----- 
 0   EventName       244126 non-null  object
 1   DeviceIDHash    244126 non-null  int64 
 2   EventTimestamp  244126 non-null  int64 
 3   ExpId           244126 non-null  int64 
dtypes: int64(3), object(1)
memory usage: 7.5+ MB

Take a closer look at the individual columns

events_data_orig['EventName'].value_counts(dropna = False)

MainScreenAppear           119205
OffersScreenAppear          46825
CartScreenAppear            42731
PaymentScreenSuccessful     34313
Tutorial                     1052
Name: EventName, dtype: int64

events_data_orig['ExpId'].value_counts(dropna = False)

248    85747
246    80304
247    78075
Name: ExpId, dtype: int64

events_data_orig['DeviceIDHash'].nunique()

7551

Reviewer's comment So,basic inoformation is here.Let's start!

Step 1. Conclusion

Open the data file and read the general information

Actions Performed

• opened and saved original data as a dataframe
  • the csv is separted by \t
• examined general info
  • 4 columns
  • 244126 entries
  • eventTimestamp column is in unix timestamp
• took a closer look at the individual columns
  • 5 different event types
  • 3 different experiment ids
  • 7551 unique users

Step 2. Prepare the data for analysis

Plan of Action

• read and save working version of data
• 2.a Rename the columns in a way that's convenient for you
• 2.b Check for missing values and data types. Correct the data if needed
• 2.c Check for and process duplicate rows
• 2.d Add a date and time column and a separate column for dates

Read and save working version of data

events_data = pd.read_csv('datasets/logs_exp_us.csv', delimiter = "\t")

2.a Rename the columns in a way that's convenient for you

events_data.columns

Index(['EventName', 'DeviceIDHash', 'EventTimestamp', 'ExpId'], dtype='object')

events_data = events_data.rename(columns = {'EventName': 'event_name',
                                           'DeviceIDHash': 'user_id', 
                                           'EventTimestamp': 'event_datetime', 
                                           'ExpId': 'exp_id'})
events_data.columns

Index(['event_name', 'user_id', 'event_datetime', 'exp_id'], dtype='object')

2.b Check for missing values and data types. Correct the data if needed

# identify missing values
print('number of missing values in dataframe')
events_data.isnull().sum()

number of missing values in dataframe

event_name        0
user_id           0
event_datetime    0
exp_id            0
dtype: int64

2.b conclusion

• None of the columns have missing values

2.c Check for and process duplicate rows

# identify duplicate rows
num_duplicates = events_data.duplicated().sum()
print('{} duplicate rows'.format(num_duplicates))

413 duplicate rows

# examine duplicate rows
duplicate_rows = events_data[events_data.duplicated()]
duplicate_rows.head(20).sort_values(by = 'user_id')

	event_name	user_id	event_datetime	exp_id
3573	MainScreenAppear	434103746454591587	1564628377	248
2350	CartScreenAppear	1694940645335807244	1564609899	248
14333	CartScreenAppear	1807104407388801321	1564655427	248
9179	MainScreenAppear	2230705996155527339	1564646087	246
4803	MainScreenAppear	2835328739789306622	1564634641	248
15746	PaymentScreenSuccessful	2877433916175408776	1564658001	247
15751	PaymentScreenSuccessful	3528217211962170139	1564658003	247
15752	PaymentScreenSuccessful	3528217211962170139	1564658003	247
15753	PaymentScreenSuccessful	3528217211962170139	1564658003	247
4076	MainScreenAppear	3761373764179762633	1564631266	247
5641	CartScreenAppear	4248762472840564256	1564637764	248
12454	PaymentScreenSuccessful	5152160705477623487	1564652139	247
9311	MainScreenAppear	5496043151846125970	1564646283	248
453	MainScreenAppear	5613408041324010552	1564474784	248
13055	PaymentScreenSuccessful	6258460144399027762	1564653224	247
5875	PaymentScreenSuccessful	6427012997733591237	1564638452	248
9990	PaymentScreenSuccessful	7035352794299231933	1564647627	248
7249	OffersScreenAppear	7224691986599895551	1564641846	246
15304	PaymentScreenSuccessful	8125832085431322921	1564657270	246
8065	CartScreenAppear	8189122927585332969	1564643929	248

perc_duplicates = num_duplicates / len(events_data)
print('{:.2%} of the data are dupicates'.format(perc_duplicates))

0.17% of the data are dupicates

# duplicates per experiment
exp_dups = duplicate_rows['exp_id'].value_counts(dropna = False).reset_index()
exp_dups.columns = ['exp_id', 'num_dups']
exp_dups

	exp_id	num_dups
0	248	165
1	247	125
2	246	123

# total events per exeriment
group_exp = events_data['exp_id'].value_counts(dropna = False).reset_index()
group_exp.columns = ['exp_id', 'total_exp']
group_exp

	exp_id	total_exp
0	248	85747
1	246	80304
2	247	78075

# combine tables
group_dup_perc = pd.merge(group_exp, exp_dups, on = 'exp_id')
group_dup_perc

	exp_id	total_exp	num_dups
0	248	85747	165
1	246	80304	123
2	247	78075	125

group_dup_perc['perc_dup'] = (group_dup_perc['num_dups'] / group_dup_perc['total_exp'] * 100).round(2)
group_dup_perc

	exp_id	total_exp	num_dups	perc_dup
0	248	85747	165	0.19
1	246	80304	123	0.15
2	247	78075	125	0.16

2c conclusion

• there are 413 duplicate rows
• this makes up 0.17% of the total data
  • the percentage of duplicates per experiment are:
    • 246: 0.15%
    • 247: 0.16%
    • 248: 0.19%
  • these values are below the general 10% rule and will be deleted

# delete duplicate rows and reset index
events_data = events_data.drop_duplicates().reset_index(drop = True)
# confirm duplicate rows are removed
print('number of duplicate rows after duplicates removed')
events_data.duplicated().sum()

number of duplicate rows after duplicates removed

0

# change unix timestamp to datetime
events_data['event_datetime'] = pd.to_datetime(events_data['event_datetime'], unit = 's')
events_data.head()

	event_name	user_id	event_datetime	exp_id
0	MainScreenAppear	4575588528974610257	2019-07-25 04:43:36	246
1	MainScreenAppear	7416695313311560658	2019-07-25 11:11:42	246
2	PaymentScreenSuccessful	3518123091307005509	2019-07-25 11:28:47	248
3	CartScreenAppear	3518123091307005509	2019-07-25 11:28:47	248
4	PaymentScreenSuccessful	6217807653094995999	2019-07-25 11:48:42	248

events_data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 243713 entries, 0 to 243712
Data columns (total 4 columns):
 #   Column          Non-Null Count   Dtype         
---  ------          --------------   -----         
 0   event_name      243713 non-null  object        
 1   user_id         243713 non-null  int64         
 2   event_datetime  243713 non-null  datetime64[ns]
 3   exp_id          243713 non-null  int64         
dtypes: datetime64[ns](1), int64(2), object(1)
memory usage: 7.4+ MB

2.d Add a date and time column and a separate column for dates

events_data['event_date'] = events_data['event_datetime'].astype('datetime64[D]')
events_data['event_time'] = events_data['event_datetime'].dt.time
events_data.head()

	event_name	user_id	event_datetime	exp_id	event_date	event_time
0	MainScreenAppear	4575588528974610257	2019-07-25 04:43:36	246	2019-07-25	04:43:36
1	MainScreenAppear	7416695313311560658	2019-07-25 11:11:42	246	2019-07-25	11:11:42
2	PaymentScreenSuccessful	3518123091307005509	2019-07-25 11:28:47	248	2019-07-25	11:28:47
3	CartScreenAppear	3518123091307005509	2019-07-25 11:28:47	248	2019-07-25	11:28:47
4	PaymentScreenSuccessful	6217807653094995999	2019-07-25 11:48:42	248	2019-07-25	11:48:42

events_data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 243713 entries, 0 to 243712
Data columns (total 6 columns):
 #   Column          Non-Null Count   Dtype         
---  ------          --------------   -----         
 0   event_name      243713 non-null  object        
 1   user_id         243713 non-null  int64         
 2   event_datetime  243713 non-null  datetime64[ns]
 3   exp_id          243713 non-null  int64         
 4   event_date      243713 non-null  datetime64[ns]
 5   event_time      243713 non-null  object        
dtypes: datetime64[ns](2), int64(2), object(2)
memory usage: 11.2+ MB

Step 2. Conclusion

Prepare the data for analysis

Actions Performed

• read and saved working version of data
• 2.a Renamed the columns
• 2.b Checked for missing values and data types. Correct the data if needed
  • 0 missing values found
• 2.c Checked for and processed duplicate rows
  • 413 duplicates found
    • equals 0.17% of total data
      • 246: 0.15%
      • 247: 0.16%
      • 248: 0.19%
    • these values are below the general 10% rule and were deleted
    • duplicates may have occured because due to data acquisition error
  • changed unix timestame to datetime to be more readable/user friendly
• 2.d Added 2 new columns for date and time

Reviewer's comment Excellent preparation! We have done all and ready! :)

Step 3. Study and check the data

• 3.a How many events are in the logs?
• 3.b How many users are in the logs?
• 3.c What's the average number of events per user?
• 3.d What period of time does the data cover?
  • 3.d.1 Find the maximum and the minimum date.
  • 3.d.2 Plot a histogram by date and time.
  • 3.d.3 Find the moment at which the data starts to be complete and ignore the earlier section.
    • Can you be sure that you have equally complete data for the entire period?
    • Older events could end up in some users' logs for technical reasons, and this could skew the overall picture.
  • 3.d.4 What period does the data actually represent?
• 3.e Did you lose many events and users when excluding the older data?
• 3.f Make sure you have users from all three experimental groups.

# review data
events_data.head(2)

	event_name	user_id	event_datetime	exp_id	event_date	event_time
0	MainScreenAppear	4575588528974610257	2019-07-25 04:43:36	246	2019-07-25	04:43:36
1	MainScreenAppear	7416695313311560658	2019-07-25 11:11:42	246	2019-07-25	11:11:42

3.a How many events are in the logs?

# calculate the number of entries
num_events = len(events_data)
print('There are {} events in the duplicate free data. \n'.format(num_events))

# create a table with counts of each experiments events
events_per_exp =  events_data.pivot_table(index = 'exp_id', columns = 'event_name', 
                                          values = 'user_id', aggfunc = 'count',
                                         margins = True, margins_name = 'TotalEvents')
print('Table: Number of events per experiment and number of event types per experiment')
events_per_exp

There are 243713 events in the duplicate free data. 

Table: Number of events per experiment and number of event types per experiment

event_name	CartScreenAppear	MainScreenAppear	OffersScreenAppear	PaymentScreenSuccessful	Tutorial	TotalEvents
exp_id
246	14798	38249	14904	11912	318	80181
247	12548	39677	15341	10039	345	77950
248	15322	41175	16563	12167	355	85582
TotalEvents	42668	119101	46808	34118	1018	243713

3.b How many users are in the logs?

# calculate the number of unique users
num_users = events_data['user_id'].nunique()
print('There are {} unique users in the duplicate free data.\n'.format(num_users))

# create a table of the number of users per group
users_per_exp = events_data.groupby('exp_id').agg({'user_id': 'nunique'}).reset_index()
users_per_exp.columns = ['exp_id', 'num_users']
print('Table: Number of users per experiment')
users_per_exp

There are 7551 unique users in the duplicate free data.

Table: Number of users per experiment

	exp_id	num_users
0	246	2489
1	247	2520
2	248	2542

3.c What's the average number of events per user?

# calculate the average number of events per user
num_events_per_user = events_data.groupby('user_id')['event_name'].count().mean().round(0)
print('The average number of events per user is {}'.format(num_events_per_user))

The average number of events per user is 32.0

Reviewer's comment Good numbers! Thank you for the details! :)

3.d What period of time does the data cover?

• 3.d.1 Find the maximum and the minimum date.
• 3.d.2 Plot a histogram by date and time.
• 3.d.3 Find the moment at which the data starts to be complete and ignore the earlier section.
  • Can you be sure that you have equally complete data for the entire period?
  • Older events could end up in some users' logs for technical reasons, and this could skew the overall picture.
• 3.d.4 What period does the data actually represent?

3.d.1 Find the maximum and the minimum date.

# find the min and max dates
min_date = events_data['event_date'].min().date()
max_date = events_data['event_date'].max().date()
print('The data covers dates from {} to {}.'.format(min_date, max_date))

The data covers dates from 2019-07-25 to 2019-08-07.

3.d.2 Plot a histogram by date and time.

# find the number of events per date
events_per_date = events_data.groupby('event_date').agg({'exp_id': 'count'}).reset_index()
events_per_date.columns = ['event_date', 'num_events_per_date']
events_per_date

	event_date	num_events_per_date
0	2019-07-25	9
1	2019-07-26	31
2	2019-07-27	55
3	2019-07-28	105
4	2019-07-29	184
5	2019-07-30	412
6	2019-07-31	2030
7	2019-08-01	36141
8	2019-08-02	35554
9	2019-08-03	33282
10	2019-08-04	32968
11	2019-08-05	36058
12	2019-08-06	35788
13	2019-08-07	31096

3.d.3 Find the moment at which the data starts to be complete and ignore the earlier section.

# plot the results
plt.figure(figsize = (12, 6))
events_per_date_graph = sns.barplot(data = events_per_date, x = events_per_date['event_date'].dt.date, y = 'num_events_per_date') #, 

plt.title('Number of events per date', size = 15)
plt.xlabel('date', size = 15)
plt.xticks(rotation = 60)
plt.ylabel('number of events', size = 15)

for bar in events_per_date_graph.patches:
    events_per_date_graph.annotate(format(bar.get_height(), '.0f'), 
                xy=(bar.get_x() + bar.get_width() / 2, bar.get_height()),
                ha = 'center', 
                va = 'center', 
                xytext = (0, 7), 
                textcoords = 'offset points')
    
plt.show()

3.d.4 What period does the data actually represent?

• The data covers 2019-07-25 to 2019-07-31, which is 14 days of data
• From 07-25 to 07-30, there is very little data
  • The first week of data may have low event numbers because of incorrect data collection and/or aggregation or missing data
• Ultimately there is sufficient data to cover 2019-08-01 to 2019-08-07, which is the last 7 days of the data

Reviewer's comment Yes, August the 1st is good variant!

Reviewer's comment Also we can leave last 3 hours of 31 of July. And add a graph for new period. :)

Student answer. Oh! I didnt think of that, thank you!

Reviewer's comment. V.2. At job we need to check this 3 hours. What type of events there? Why? etc :)

3.e Did you lose many events and users when excluding the older data?

# create a new table excluding the dates with minimal data
events_filtered = events_data.query('event_date > "2019-07-31"').reset_index(drop = True)
events_filtered['event_date'].value_counts()

2019-08-01    36141
2019-08-05    36058
2019-08-06    35788
2019-08-02    35554
2019-08-03    33282
2019-08-04    32968
2019-08-07    31096
Name: event_date, dtype: int64

# filtered data
# calculate the number of entries
filtered_num_events = len(events_filtered)
print('There are {} events in the filtered data. \n'.format(filtered_num_events))

# create a table with counts of each experiments events
filtered_events_per_exp =  events_filtered.pivot_table(index = 'exp_id', columns = 'event_name', 
                                          values = 'user_id', aggfunc = 'count',
                                         margins = True, margins_name = 'TotalEventsFiltered')

print('Table: Number of events per experiment and number of event types per experiment in filtered data')
filtered_events_per_exp

There are 240887 events in the filtered data. 

Table: Number of events per experiment and number of event types per experiment in filtered data

event_name	CartScreenAppear	MainScreenAppear	OffersScreenAppear	PaymentScreenSuccessful	Tutorial	TotalEventsFiltered
exp_id
246	14690	37676	14767	11852	317	79302
247	12434	39090	15179	9981	338	77022
248	15179	40562	16387	12085	350	84563
TotalEventsFiltered	42303	117328	46333	33918	1005	240887

# unfiltered data
print('Table: Number of events per experiment and number of event types per experiment (unfiltered data)')
events_per_exp

Table: Number of events per experiment and number of event types per experiment (unfiltered data)

event_name	CartScreenAppear	MainScreenAppear	OffersScreenAppear	PaymentScreenSuccessful	Tutorial	TotalEvents
exp_id
246	14798	38249	14904	11912	318	80181
247	12548	39677	15341	10039	345	77950
248	15322	41175	16563	12167	355	85582
TotalEvents	42668	119101	46808	34118	1018	243713

#calculate the difference between filtered and unfiltered events
event_diff = len(events_data) - filtered_num_events

print('The unfitered data has {} entries, the filtered data has {} entries.\nThis is a difference of {} entries and accounts for {:.1%} of the unfiltered data'.format(len(events_data), len(events_filtered), event_diff, (event_diff / len(events_data))))

The unfitered data has 243713 entries, the filtered data has 240887 entries.
This is a difference of 2826 entries and accounts for 1.2% of the unfiltered data

3.e Conclusion

Did you lose many events and users when excluding the older data?

• The unfitered data has 243713 entries, the filtered data has 240887 entries.
• This is a difference of 2826 entries and accounts for 1.2% of the unfiltered data
  • this falls under 10% of the data and so removing will not affet the results

Reviewer's comment So, it is possible numbers. Good choice! :)

3.f Make sure you have users from all three experimental groups.

# calculate the number of unique users in the filtered data
filtered_num_users = events_filtered['user_id'].nunique()
print('There are {} unique users in the filtered data.\n'.format(filtered_num_users))

# create a table of the number of users per group in the filtered data
filtered_users_per_exp = events_filtered.groupby('exp_id').agg({'user_id': 'nunique'}).reset_index()
filtered_users_per_exp.columns = ['exp_id', 'filtered_num_users']
print('Table: Number of users per experiment in filtered data')
filtered_users_per_exp

There are 7534 unique users in the filtered data.

Table: Number of users per experiment in filtered data

	exp_id	filtered_num_users
0	246	2484
1	247	2513
2	248	2537

# unfiltered data
print('There are {} unique users in the duplicate free data.\n'.format(num_users))

print('Table: Number of users per experiment (unfiltered data)')
users_per_exp

There are 7551 unique users in the duplicate free data.

Table: Number of users per experiment (unfiltered data)

	exp_id	num_users
0	246	2489
1	247	2520
2	248	2542

# determine the percent of users filtered out per experiment
filtered_users_per_exp['perc_removed_users'] = ((1 - (filtered_users_per_exp['filtered_num_users'] / users_per_exp['num_users'])) * 100).round(2)
filtered_users_per_exp

	exp_id	filtered_num_users	perc_removed_users
0	246	2484	0.20
1	247	2513	0.28
2	248	2537	0.20

user_diff = num_users - filtered_num_users
print('The unfiltered data has {} number of users, the filtered data has {}.\nThis is a difference of {} and accounts for {:.1%} of the unfiltered users'.format(num_users, filtered_num_users, user_diff, (user_diff / num_users)))
print('The filtered users per experiment group account for {} percent per group'.format(list(filtered_users_per_exp['perc_removed_users'])))

The unfiltered data has 7551 number of users, the filtered data has 7534.
This is a difference of 17 and accounts for 0.2% of the unfiltered users
The filtered users per experiment group account for [0.2, 0.28, 0.2] percent per group

3.f Conclusion

Make sure you have users from all three experimental groups.

• The unfiltered data has 7551 number of users, the filtered data has 7534.
• This is a difference of 17 and accounts for 0.2% of the unfiltered users
• The filtered users per experiment group account for 0.2%, 0.28%, and 0.2% per group
  • this falls under 10% of the data and so removing will not affet the results

Reviewer's comment Groups are normal. Let's continue! :)

Step 3. Conclusion

Study and check the data

After removing duplicates

• 243713 events in the logs
• 7551 unique users in 3 exeriments
  • experiment # 246: 2489
  • experiment # 247: 2520
  • experiment # 248: 2542
• on average there are 32 events per user
• The experiments cover data for 14 days from 2019-07-25 to 2019-08-07
  • There is sufficient data for 7 days from 2019-08-01 through 2019-08-07
    • Insufficient entries in the first week (2019-07-25 - 2019-07-31) may be due to incorrect data collection and/or aggregation, or missing data

After removing insufficient data and focusing on events from 2019-08-01 to 2019-08-07

• 240887 entries remain
  • this is a difference of 2826 entries
  • 1.2% of the data was filtered out
• 7534 unique users remain
  • This is a difference of 17 users
  • 0.2% of the users were filtered out
    • 0.28% or less of users were filtered out per experiment

Reviewer's comment Thank you for conclusion after step!

Step 4. Study the event funnel

• 4.a See what events are in the logs and their frequency of occurrence. Sort them by frequency.
• 4.b Find the number of users who performed each of these actions. Sort the events by the number of users.
  • 4.b.1 Calculate the proportion of users who performed the action at least once.
• 4.c In what order do you think the actions took place. Are all of them part of a single sequence? You don't need to take them into account when calculating the funnel.
• 4.d Use the event funnel to find the share of users that proceed from each stage to the next. (For instance, for the sequence of events A → B → C, calculate the ratio of users at stage B to the number of users at stage A and the ratio of users at stage C to the number at stage B.)
• 4.e At what stage do you lose the most users?
• 4.f What share of users make the entire journey from their first event to payment?

# review data
events_filtered.head(2)

	event_name	user_id	event_datetime	exp_id	event_date	event_time
0	Tutorial	3737462046622621720	2019-08-01 00:07:28	246	2019-08-01	00:07:28
1	MainScreenAppear	3737462046622621720	2019-08-01 00:08:00	246	2019-08-01	00:08:00

4.a See what events are in the logs and their frequency of occurrence. Sort them by frequency.

# find the number of events per type
event_counts = events_filtered.groupby('event_name').agg({'exp_id': 'count'}).reset_index()
event_counts.columns = ['event_name', 'num_events_per_type']
event_counts

	event_name	num_events_per_type
0	CartScreenAppear	42303
1	MainScreenAppear	117328
2	OffersScreenAppear	46333
3	PaymentScreenSuccessful	33918
4	Tutorial	1005

# find the percentages of each event type
event_counts['perc'] = (event_counts['num_events_per_type'] / len(events_filtered) * 100).round(2)
event_counts.sort_values(by = 'num_events_per_type', ascending = False)

	event_name	num_events_per_type	perc
1	MainScreenAppear	117328	48.71
2	OffersScreenAppear	46333	19.23
0	CartScreenAppear	42303	17.56
3	PaymentScreenSuccessful	33918	14.08
4	Tutorial	1005	0.42

# plot the results
plt.figure(figsize = (12, 6))
event_counts_graph = sns.barplot(data = event_counts, x = 'event_name', y = 'num_events_per_type', 
                                order = event_counts.sort_values(by = 'num_events_per_type', ascending = False).event_name)
plt.title('Number of Events per Type', size = 15)
plt.xlabel('event type', size = 15)
plt.ylabel('number of events')
plt.xticks(rotation = 30)

for bar in event_counts_graph.patches:
    event_counts_graph.annotate(format(bar.get_height(), '.0f'), 
                xy=(bar.get_x() + bar.get_width() / 2, bar.get_height()),
                ha = 'center', 
                va = 'center', 
                xytext = (0, 7), 
                textcoords = 'offset points')
plt.show()

4.a Conclusion

See what events are in the logs and their frequency of occurrence. Sort them by frequency.

• MainScreenAppear has the most events at 117328 and account for 49% of events
• followed by OffersScreenAppear, CartScreenAppear, PaymentScreenSuccessful with 46333, 42303, and 33918 respectively and accounting for 19%, 17%, and 14% respectively
• Tutorial has the least at 1005
  • this accounts for 0.4% of events, suggesting that the app is user friendly for most and is not a requirement for users

Reviewer's comment So, we sort events by frequency. And we know procents. Thay are not so good for us.

4.b Find the number of users who performed each of these actions. Sort the events by the number of users.

# review data
events_filtered.head(2)

	event_name	user_id	event_datetime	exp_id	event_date	event_time
0	Tutorial	3737462046622621720	2019-08-01 00:07:28	246	2019-08-01	00:07:28
1	MainScreenAppear	3737462046622621720	2019-08-01 00:08:00	246	2019-08-01	00:08:00

# create a table to find the number of unique users per event
users_per_event = events_filtered.groupby('event_name').agg({'user_id': 'nunique'}).reset_index()
users_per_event.columns = ['event_name', 'num_unique_users']
users_per_event['perc_users'] = (users_per_event['num_unique_users'] / events_filtered['user_id'].nunique() * 100).round()
users_per_event.sort_values(by = 'num_unique_users')

	event_name	num_unique_users	perc_users
4	Tutorial	840	11.0
3	PaymentScreenSuccessful	3539	47.0
0	CartScreenAppear	3734	50.0
2	OffersScreenAppear	4593	61.0
1	MainScreenAppear	7419	98.0

# plot the counts
plt.figure(figsize = (12, 6))
users_per_event_graph = sns.barplot(data = users_per_event, x = 'event_name', y = 'num_unique_users',
                                   order = users_per_event.sort_values(by = 'num_unique_users', ascending = False).event_name)

plt.title('Number of Unique Users per Event Type', size = 15)
plt.xlabel('event type', size = 15)
plt.ylabel('number of unique users')
plt.xticks(rotation = 30)

for bar in users_per_event_graph.patches:
    users_per_event_graph.annotate(format(bar.get_height(), '.0f'), 
                xy=(bar.get_x() + bar.get_width() / 2, bar.get_height()),
                ha = 'center', 
                va = 'center', 
                xytext = (0, 7), 
                textcoords = 'offset points')
plt.show()

4.b Conclusion

Find the number of users who performed each of these actions. Sort the events by the number of users.

• The number of unique users per event is representative of the total data
  • MainScreenAppear has the most users (7419), followed by OfferScreenAppear (4593), CartScreenAppear (3734), PaymentScreenSuccessful (3539), Tutorial (840)
• This data is suggestive of a user funnel where they see the Main screen, then an offer screen, add something to the cart and make a payment. Tutorial may be optional
  • The percent of users goes down as you move through this funnel
    • 98% to 61% to 50% to 47% to 11%

Reviewer's comment Unique numbers show better procennts! :)

4.b.1 Calculate the proportion of users who performed the action at least once.

# review data
events_filtered.head(2)

	event_name	user_id	event_datetime	exp_id	event_date	event_time
0	Tutorial	3737462046622621720	2019-08-01 00:07:28	246	2019-08-01	00:07:28
1	MainScreenAppear	3737462046622621720	2019-08-01 00:08:00	246	2019-08-01	00:08:00

# determine what actions a user performed at least once
user_event_breakdown = events_filtered.groupby(['user_id', 'event_name']).agg({'event_date': 'nunique'})
user_event_breakdown.head(15)

		event_date
user_id	event_name
6888746892508752	MainScreenAppear	1
6909561520679493	CartScreenAppear	1
	MainScreenAppear	1
	OffersScreenAppear	1
	PaymentScreenSuccessful	1
6922444491712477	CartScreenAppear	3
	MainScreenAppear	3
	OffersScreenAppear	3
	PaymentScreenSuccessful	3
7435777799948366	MainScreenAppear	2
7702139951469979	CartScreenAppear	4
	MainScreenAppear	7
	OffersScreenAppear	7
	PaymentScreenSuccessful	4
8486814028069281	CartScreenAppear	1

# count the number of actions perfromed by an individual user
user_actions = events_filtered.groupby('user_id').agg({'event_name': 'nunique'}).reset_index()
user_actions.columns = ['user_id', 'num_actions_performed']
user_actions.head()

	user_id	num_actions_performed
0	6888746892508752	1
1	6909561520679493	4
2	6922444491712477	4
3	7435777799948366	1
4	7702139951469979	4

# count the number of users that performed the actions
action_proportions = user_actions.groupby('num_actions_performed').agg({'user_id': 'count'}).reset_index()
action_proportions.columns = ['num_actions_performed', 'num_users']
action_proportions

	num_actions_performed	num_users
0	1	2717
1	2	1004
2	3	318
3	4	3029
4	5	466

# create a new column with proportions of users who performed the actions
action_proportions['percentage_performed'] = (action_proportions['num_users'] / len(user_actions) * 100).round(2)
action_proportions

	num_actions_performed	num_users	percentage_performed
0	1	2717	36.06
1	2	1004	13.33
2	3	318	4.22
3	4	3029	40.20
4	5	466	6.19

# plot the results
plt.figure(figsize = (7, 7))
plt.pie(x = action_proportions['percentage_performed'], labels = action_proportions['num_actions_performed'],
       autopct = '%.0f%%', textprops={'fontsize': 15})
plt.title('Percentage of Number of Actions Performed by Individual Users', size = 15)
plt.show()

4.b.1 Conclusion

Calculate the proportion of users who performed the action at least once.

Percentage of Number of Actions Performed by Individual Users

• 40% of individual users performed 4 out of 5 actions
  • although there are 5 actions, one of them is a tutorial, which the data indicates is not a required action
  • so perhaps more accurately 40% of users complete 100% of the required actions
• 36% complete only one action
• 13% complete two actions, and 4% complete 3, and 6% complete all 5 actions
• What is interesting about this data is that there are nearly equal amounts of users completing 1 and 4 of the actions

Reviewer's comment Very intresting information! Now we can check users actions!

4.c In what order do you think the actions took place. Are all of them part of a single sequence? You don't need to take them into account when calculating the funnel.

4.c Conclusion

In what order do you think the actions took place. Are all of them part of a single sequence? You don't need to take them into account when calculating the funnel.

• Based on Number of Events per Type & Number of Unique Users per Event Type graphs, the order of actions is likely:

  • 1. MainScreenAppear
  • 1. OfferScreenAppear
  • 1. CartScreenAppear
  • 1. PaymentScreenSuccessful
  • Tutorial is optional

• While there may be a standard pattern followed by most users, I dont think the events must happen in this order.

  • For example, when each user's actions are broken down and then counted by number of datetimes, some users, such as user # 7702139951469979, sees the OfferScreenAppear 2 times more than the MainScreenAppear, suggesting they are not going in the above order but may be seeing the OfferScreenAppear first and then the MainScreenAppear.

Reviewer's comment Great conclusion!

4.d Use the event funnel to find the share of users that proceed from each stage to the next. (For instance, for the sequence of events A → B → C, calculate the ratio of users at stage B to the number of users at stage A and the ratio of users at stage C to the number at stage B.)

# create a table focusing on required funnel
funnel = users_per_event.query('event_name != "Tutorial"')
funnel

	event_name	num_unique_users	perc_users
0	CartScreenAppear	3734	50.0
1	MainScreenAppear	7419	98.0
2	OffersScreenAppear	4593	61.0
3	PaymentScreenSuccessful	3539	47.0

# order the users per event
funnel.sort_values(by = 'perc_users', ascending = False).reset_index(drop = True)

	event_name	num_unique_users	perc_users
0	MainScreenAppear	7419	98.0
1	OffersScreenAppear	4593	61.0
2	CartScreenAppear	3734	50.0
3	PaymentScreenSuccessful	3539	47.0

funnel['perc_change'] = round(funnel['num_unique_users'].sort_values(ascending = False).pct_change(periods = 1) * 100, 0)
funnel = funnel.fillna(0)
funnel.sort_values(by = 'num_unique_users', ascending = False)

	event_name	num_unique_users	perc_users	perc_change
1	MainScreenAppear	7419	98.0	0.0
2	OffersScreenAppear	4593	61.0	-38.0
0	CartScreenAppear	3734	50.0	-19.0
3	PaymentScreenSuccessful	3539	47.0	-5.0

4.d Conclusion

Use the event funnel to find the share of users that proceed from each stage to the next. (For instance, for the sequence of events A → B → C, calculate the ratio of users at stage B to the number of users at stage A and the ratio of users at stage C to the number at stage B.)

• From the MainScreenAppear to OfferScreenAppear, there is a 38% reduction in users
  • 38% of users leave before making it to the OfferScreenAppear
  • maybe they didnt find what they are looking for or didnt like the prices
• From OfferScreenAppear to CartScreenAppear there is a 19% reduction in users
  • 19% of users leave before adding something to the cart after seeing the offer page
  • maybe they didnt find what they are looking for, didnt like the prices, or couldnt figure out how to add to the cart
• from CartScreenAppear to PaymentScreenSuccessful, there is a 5% reduction
  • most users who get to the cart, also make a successful order
  • only 5% decide not to make a purchase once they get to the cart
    • maybe they couldnt because of the app malfunctioning or they decided to wait to order

Reviewer's comment Thank you for such wonderful conclusion!

Reviewer's comment Could you plot a graphs for funnel? :)

Student answer. Graph added. This is a very useful way to look at funnel data, thank you!

Reviewer's comment. V.2. Perfect graph with procents! :)

Reviewer's comment. V.2. Also we can plot the same graph for groups. It give us more information. :)

# reset index with sorted values
funnel_sort = funnel.sort_values(by = 'num_unique_users', ascending = False).reset_index()
funnel_sort

	index	event_name	num_unique_users	perc_users	perc_change
0	1	MainScreenAppear	7419	98.0	0.0
1	2	OffersScreenAppear	4593	61.0	-38.0
2	0	CartScreenAppear	3734	50.0	-19.0
3	3	PaymentScreenSuccessful	3539	47.0	-5.0

# plot a funnel chart
funnel_chart = go.Figure(go.Funnel(y = list(funnel_sort['event_name']), 
                                  x = list(funnel_sort['num_unique_users'])))
funnel_chart.update_layout(title = 'Sales/Event Funnel Chart')
funnel_chart.show()

4.e At what stage do you lose the most users?

4.e Conclusion

At what stage do you lose the most users?

• Most users are lost between the MainScreenAppear and OffersScreenAppear: 38%
  • maybe users arent finding what they are looking for or the prices are too high

Reviewer's comment You are right!

4.f What share of users make the entire journey from their first event to payment?

4.f Conclusion

What share of users make the entire journey from their first event to payment?

• 47% of users go through the entire funnel
• this is a little bit less than half of the users
  • maybe the site attracts users who arent finding what theyre looking for

Step 4. Conclusion

Study the event funnel

• Based on the number of unique users per event and the percent change from one event to the next, there likely funnel order is:
  • 1. MainScreenAppear (98% unique users)
  • 1. OfferScreenAppear (61% unique users, a 38% reduction)
  • 1. CartScreenAppear (50% unique users, a 19% reduction)
  • 1. PaymentScreenSuccessful (47% unique users, a 5% reduction)
  • Tutorial is optional (11% unique users)
• The most users are lost from the MainScreenAppear to the OfferScreenAppear: 38%
• 95% of users who make it to the CartScreenAppear also make it to the PaymentScreenSuccessful
  • This suggests that once a user puts an item in the cart, they are very likely to make a purchase
• 47% of users go through the entire funnel (all four required events)
• Nearly equal are users that performed
  • 1 action (36%)
    • likely MainScreenAppear
  • 4 actions (40%)
    • likely the whole funnel

Reviewer's comment Yep!

Step 5. Study the results of the experiment

• 5.a How many users are there in each group?
• 5.b Determine if there is a statistically significant difference between sample groups
  • 5.b.1 Create a function to find the following for each action in the funnel
    • find the number of users who performed the action
    • find the share of users who performed the action
    • test if the groups are statistically significant different
  • 5.b.2 Use the function to perform:
    • A/A test
      • A/A test conclusion](#aatestconclusion)
    • A1/B test
      • A1/B test conclusion
    • A2/B test
      • A2/B test conclusion
    • A combo/B test
      • A combo/B conclusion
• 5.c What significance level have you set to test the statistical hypotheses mentioned above?
  • 5.c.1 Calculate how many statistical hypothesis tests you carried out.)
  • 5.c.2 With a statistical significance level of 0.1, one in 10 results could be false. What should the significance level be?
  • 5.c.3 If you want to change it, run through the previous steps again and check your conclusions.

5.a How many users are there in each group?

# review data
events_filtered.head(2)

	event_name	user_id	event_datetime	exp_id	event_date	event_time
0	Tutorial	3737462046622621720	2019-08-01 00:07:28	246	2019-08-01	00:07:28
1	MainScreenAppear	3737462046622621720	2019-08-01 00:08:00	246	2019-08-01	00:08:00

# count the number of users per group
exp_groups = events_filtered.groupby('exp_id').agg({'user_id': 'nunique'}).reset_index()
exp_groups.columns = ['exp_id', 'users_per_group']
exp_groups

	exp_id	users_per_group
0	246	2484
1	247	2513
2	248	2537

exp_perc_diff = 1 - exp_groups['users_per_group'].min() / exp_groups['users_per_group'].max()
print('There is a {:.1%} difference between exp 246 (least users) and exp 248 (most users)'.format(exp_perc_diff))

There is a 2.1% difference between exp 246 (least users) and exp 248 (most users)

Reviewer's comment Numbers are ok. :)

5.a Conclusion

How many users are there in each group?

• exp 246 has 2484 users
• exp 247 has 2513 users
• exp 248 has 2537 users
• experiment groups are roughly the same size, with a 2.1% or less difference between the groups

5.b Determine if there is a statistically significant difference between sample groups

• Here are the criteria for a successful A/A test:
  • The number of users in different groups doesn't vary by more than 1%
  • For all groups, data on the same event is recorded and sent to analytical systems
  • None of the key metrics varies by a statistically significant amount—typically by no more than 1%
  • Users remain within their groups until the end of the test. If they see different versions of the page during the study, it won't be clear which version influenced their decisions, so the reliability of the results will be compromised

# determine if there are users in more than one group
users_test = events_filtered.groupby('user_id').agg({'exp_id': 'nunique'})
users_test.head()

	exp_id
user_id
6888746892508752	1
6909561520679493	1
6922444491712477	1
7435777799948366	1
7702139951469979	1

users_query = users_test.query('exp_id > 1')
users_query

	exp_id
user_id

• There are no users in multiple groups

# determine the percent difference between groups
exp_groups['perc_diff'] = round(exp_groups['users_per_group'].pct_change(periods = 1) * 100, 3)
exp_groups

	exp_id	users_per_group	perc_diff
0	246	2484	NaN
1	247	2513	1.167
2	248	2537	0.955

• The number of users in the two control groups 246 and 246 varies by about 1%, putting in the appropriate range for testing.

# prepare a table for hypothesis testing the experimental groups and combinations
experiments = events_filtered.pivot_table(index = 'event_name', columns = 'exp_id',
                                         values = 'user_id', aggfunc = 'nunique').reset_index()
experiments.columns = ['event_name', '246', '247', '248']
experiments

	event_name	246	247	248
0	CartScreenAppear	1266	1238	1230
1	MainScreenAppear	2450	2476	2493
2	OffersScreenAppear	1542	1520	1531
3	PaymentScreenSuccessful	1200	1158	1181
4	Tutorial	278	283	279

# create a column with the combination of the two control groups
experiments['249'] = experiments['246'] + experiments['247']
experiments

	event_name	246	247	248	249
0	CartScreenAppear	1266	1238	1230	2504
1	MainScreenAppear	2450	2476	2493	4926
2	OffersScreenAppear	1542	1520	1531	3062
3	PaymentScreenSuccessful	1200	1158	1181	2358
4	Tutorial	278	283	279	561

# create a new table to combine the count of users from the control groups
# count the number of users per group
exp_groups_counts = events_filtered.groupby('exp_id').agg({'user_id': 'nunique'}).reset_index()
exp_groups_counts.columns = ['exp_id', 'users_per_group']
exp_groups_counts

	exp_id	users_per_group
0	246	2484
1	247	2513
2	248	2537

# create a dictionary with the count of the control groups users combined
controls_combo = {'exp_id': 249, 
                  'users_per_group': exp_groups_counts[exp_groups_counts['exp_id'] == 246]['users_per_group'].iloc[0] + 
                 exp_groups_counts[exp_groups_counts['exp_id'] == 247]['users_per_group'].iloc[0]}
controls_combo

{'exp_id': 249, 'users_per_group': 4997}

# combine into one dataframe
exp_counts_combo = exp_groups_counts.append(controls_combo, ignore_index = True)
exp_counts_combo

	exp_id	users_per_group
0	246	2484
1	247	2513
2	248	2537
3	249	4997

5.b.1 Create a function to find the following for each action in the funnel

Testing the Hypothesis that Proportions Are Equal - Lesson

In stats we learned about hypothesis testing of the means of populations

- made conclusions based on samples comparing their means w/ a certain number or seeing if 2 means are equal to each other

Another typical task is to test hypotheses about the equality of proportions of populations

The difference between the proportions we observe in our samples will be our STATISTIC

Z = (P1 - P2) - (π₁ - π₂) / sqrt(P(1 - P)(1/n1 + 1/n2)) ~ N(0,1)

Z = standard value for a criterion with a standard normal distribution, where the mean is 0 and the standard deviation is 1 the expression is distributed as N(0,1)

n1, n2 = sizes of the two samples being compared ie the number of observations they contain

P1, P2 = proportions observed in the samples

P = P1 + P2

π₁, π₂ = the actual proportions in the populations being compared

With A/B testing, one usually tests the hypothesis that π₁ = π₂. Then, if the null hypothesis is true, the expression (π₁ - π₂) in the nominator will equal 0 and the criterion can be calculated using only the sample data. The statistic thus obtained will be normally distributed, making it possible to carry out two-sided and one-sided (bilateral and unilateral) tests. Using the same null hypothesis that two populations' proportions are equal, we can test the alternative hypotheses that 1) the proportions simply aren't equal, or that 2) one proportion is larger or smaller than the other.

# create a function for testing the hypotheses
def test_hypothesis(exp_a, exp_b, alpha):
    
    # iterate over each event
    for event in experiments['event_name']:
        
        # define the successes, i.e. the number of users who performed the action
        successes_a = experiments[experiments['event_name'] == event][exp_a].iloc[0]
        successes_b = experiments[experiments['event_name'] == event][exp_b].iloc[0]
        
        # define the trials, i.e. the number of users in the experiment group
        exp_group_a = exp_counts_combo[exp_counts_combo['exp_id'] == int(exp_a)]['users_per_group'].iloc[0]
        exp_group_b = exp_counts_combo[exp_counts_combo['exp_id'] == int(exp_b)]['users_per_group'].iloc[0]
        
        # proportion of success in experiment a
        p1 = successes_a / exp_group_a
        
        # proportion of success in experiment b
        p2 = successes_b / exp_group_b
        
        # success proportion in the combined dataset
        p_combined = (successes_a + successes_b) / (exp_group_a + exp_group_b)
        
        # difference between the proportions
        difference = p1 - p2
        
        # calculating the statistic in standard deviations of the standard normal distribution
        # Z = standard value for a criterion with a standard normal distribution, 
        # where the mean is 0 and the standard deviation is 1 the expression is distributed as N(0,1)
        z = difference / mth.sqrt(p_combined * (1 - p_combined) * (1/exp_group_a + 1/exp_group_b))
        
        # set up the standard normal distribution (mean 0, standard deviation 1)
        distr = st.norm(0, 1)
        
        # calculate the p-value
        p_value = (1 - distr.cdf(abs(z))) * 2
        
        print('Event: {}:'.format(event))
        print('\tp-value: ', p_value)
        
        if (p_value < alpha):
            print('\tReject the null hypothesis for {}, there is a signficant difference between the proportions of experiments {} and {}.'.format(event, exp_a, exp_b))
        else:
            print('\tFail to reject the null hypothesis for {}, there is not a signficant difference between the proportions of experiments {} and {}.'.format(event, exp_a, exp_b))
            
        # print the number of users and share of users per event for each experiment group
        print('\tExperiment:\t\t{}\t{}\n\tNumber of users:\t{}\t{}\n\tShare of users:\t\t{:.2%}\t{:.2%}'.format(exp_a, exp_b, exp_group_a, exp_group_b, p1, p2), '\n')

Reviewer's comment Excellent function and preparation! We are ready for tests! :)

5.b.2 Use the function to perform hypothesis tests

5.b.2 A/A test

Control group 246 vs control group 247

• Null Hypothesis: Experiments 246 and 247 are not statistically different
• Alternative Hypothesis: Experiments 246 and 247 are statistically different

# test control experiments 
print('control group 246 vs control group 247\n')
test_hypothesis(exp_a = '246', exp_b = '247', alpha = 0.01)

control group 246 vs control group 247

Event: CartScreenAppear:
	p-value:  0.22883372237997213
	Fail to reject the null hypothesis for CartScreenAppear, there is not a signficant difference between the proportions of experiments 246 and 247.
	Experiment:		246	247
	Number of users:	2484	2513
	Share of users:		50.97%	49.26% 

Event: MainScreenAppear:
	p-value:  0.7570597232046099
	Fail to reject the null hypothesis for MainScreenAppear, there is not a signficant difference between the proportions of experiments 246 and 247.
	Experiment:		246	247
	Number of users:	2484	2513
	Share of users:		98.63%	98.53% 

Event: OffersScreenAppear:
	p-value:  0.2480954578522181
	Fail to reject the null hypothesis for OffersScreenAppear, there is not a signficant difference between the proportions of experiments 246 and 247.
	Experiment:		246	247
	Number of users:	2484	2513
	Share of users:		62.08%	60.49% 

Event: PaymentScreenSuccessful:
	p-value:  0.11456679313141849
	Fail to reject the null hypothesis for PaymentScreenSuccessful, there is not a signficant difference between the proportions of experiments 246 and 247.
	Experiment:		246	247
	Number of users:	2484	2513
	Share of users:		48.31%	46.08% 

Event: Tutorial:
	p-value:  0.9376996189257114
	Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments 246 and 247.
	Experiment:		246	247
	Number of users:	2484	2513
	Share of users:		11.19%	11.26% 

5.b.2 A/A test Conclusion

• There is no significant difference between the two control groups
• This is an expected result. If there was a difference the data used/provided would have to be re-evaluated

Reviewer's comment It is very good! We haven't problems! :)

5.b.2 A1/B test

Control Group 246 vs Test Group 248

• Null Hypothesis: Experiments 246 and 248 are not statistically different
• Alternative Hypothesis: Experiments 246 and 248 are statistically different

# test control group vs test group 
print('control group 246 vs test group 248\n')
test_hypothesis(exp_a = '246', exp_b = '248', alpha = 0.05)

control group 246 vs test group 248

Event: CartScreenAppear:
	p-value:  0.07842923237520116
	Fail to reject the null hypothesis for CartScreenAppear, there is not a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		50.97%	48.48% 

Event: MainScreenAppear:
	p-value:  0.2949721933554552
	Fail to reject the null hypothesis for MainScreenAppear, there is not a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		98.63%	98.27% 

Event: OffersScreenAppear:
	p-value:  0.20836205402738917
	Fail to reject the null hypothesis for OffersScreenAppear, there is not a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		62.08%	60.35% 

Event: PaymentScreenSuccessful:
	p-value:  0.2122553275697796
	Fail to reject the null hypothesis for PaymentScreenSuccessful, there is not a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		48.31%	46.55% 

Event: Tutorial:
	p-value:  0.8264294010087645
	Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		11.19%	11.00% 

5.b.2 A1/B test Conclusion

Control Group 246 vs Test Group 248 Conclusion

• There is no significant difference between control group 246 and test group 248
• the font change does not affect users behavior

Reviewer's comment Well done! :)

5.b.2 A2/B test

Control Group 247 vs Test Group 248

• Null Hypothesis: Experiments 247 and 248 are not statistically different
• Alternative Hypothesis: Experiments 247 and 248 are statistically different

# test control group vs test group 
print('control group 247 vs test group 248\n')
test_hypothesis(exp_a = '247', exp_b = '248', alpha = 0.05)

control group 247 vs test group 248

Event: CartScreenAppear:
	p-value:  0.5786197879539783
	Fail to reject the null hypothesis for CartScreenAppear, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		49.26%	48.48% 

Event: MainScreenAppear:
	p-value:  0.4587053616621515
	Fail to reject the null hypothesis for MainScreenAppear, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		98.53%	98.27% 

Event: OffersScreenAppear:
	p-value:  0.9197817830592261
	Fail to reject the null hypothesis for OffersScreenAppear, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		60.49%	60.35% 

Event: PaymentScreenSuccessful:
	p-value:  0.7373415053803964
	Fail to reject the null hypothesis for PaymentScreenSuccessful, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		46.08%	46.55% 

Event: Tutorial:
	p-value:  0.765323922474501
	Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		11.26%	11.00% 

5.b.2 A2/B test Conclusion

Control Group 247 vs Test Group 248 Conclusion

• There is no significant difference between control group 247 and test group 248
• the font change does not affect users behavior

Reviewer's comment Well done again! :)

5.b.2 A combo/B test

Combined Control Group 249 (246 + 247) vs Test Group 248

• Null Hypothesis: Experiments 249 and 248 are not statistically different
• Alternative Hypothesis: Experiments 249 and 248 are statistically different

# test combined control group vs test group 
print('control groups combined 249 (246 + 247) vs test group 248\n')
test_hypothesis(exp_a = '249', exp_b = '248', alpha = 0.05)

control groups combined 249 (246 + 247) vs test group 248

Event: CartScreenAppear:
	p-value:  0.18175875284404386
	Fail to reject the null hypothesis for CartScreenAppear, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		50.11%	48.48% 

Event: MainScreenAppear:
	p-value:  0.29424526837179577
	Fail to reject the null hypothesis for MainScreenAppear, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		98.58%	98.27% 

Event: OffersScreenAppear:
	p-value:  0.43425549655188256
	Fail to reject the null hypothesis for OffersScreenAppear, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		61.28%	60.35% 

Event: PaymentScreenSuccessful:
	p-value:  0.6004294282308704
	Fail to reject the null hypothesis for PaymentScreenSuccessful, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		47.19%	46.55% 

Event: Tutorial:
	p-value:  0.764862472531507
	Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		11.23%	11.00% 

5.b.2 A combo/B test Conclusion

Combined Control Group 249 (246 + 247) vs Test Group 248 Conclusion

• There is no significant difference between the combined control groups 249 (246 + 247) and test group 248
• the font change does not affect users behavior

Reviewer's comment So, it was last and all is clear! :)

5.b Conclusion

• None of the test are statistically different, they all Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments
• This is true for all of the tests performed:
  • A/A
  • A1/B
  • A2/B
  • A combo/B
• Thus the change in fonts does not impact user behavior

Reviewer's comment Yes, fonts does not impact user behavior. :)

5.c What significance level have you set to test the statistical hypotheses mentioned above?

• For the A/A a 1% significance level was used, which is recommended for A/A tests, since they should be nearly identical if performed/collected correctly
  • For the A/B tests, a 5% significance level was used, which is standard for most types of statistical tests

5.c.1 Calculate how many statistical hypothesis tests you carried out.

• 15 statistical tests were performed
  • 5: 246 vs 248
  • 5: 247 vs 248
  • 5: 249 (246 + 247) vs 248
• The A/A (246 vs 247) is not included in the count, because they are both controls

5.c.2 With a statistical significance level of 0.1, one in 10 results could be false. What should the significance level be?

• use the Bonferroni correction to reduce the family-wise error rate (FWER), i.e. lower the chances of a false result in multiple comparisons
• the probability of at least one of a muliti-wise comparison yielding a falst positive result:
  • alpha (a) = 1 - (1-a)^k = (probability of at least one false positive * 100)%
    • Given a total number of hypotheses m = 1, FWER = ɑ
  • Bonferroni correction: alpha (a) / number of hypothesis (m) = significance level for each test

# calculate the FWER
desired_alpha = 0.05
num_tests = 15
fwer = 1 - (1 - desired_alpha) ** num_tests
print('FWER for these experiments is {}, this alpha to be used in the Bonferroni correction.'.format(fwer))
print('This means there is a {:.2%} probability of at least one false positive result out of {} tests'.format(fwer, num_tests))

FWER for these experiments is 0.536708769840247, this alpha to be used in the Bonferroni correction.
This means there is a 53.67% probability of at least one false positive result out of 15 tests

# calulate the bonferonni correction
bonferonni_correction = fwer 
print('bonferonni correction: {}'.format(bonferonni_correction))

bonferonni correction: 0.536708769840247

5.c.2 Conclusion

• The family-wise error rate (FWER) was found for the test performed
  • 15 tests were performed
  • the FWER is 0.536708769840247
  • this means there is a 53% probability of at least one false positive result in the 15 tests
  • the FWER is used as the alpha for the Bonferonni Correction to try to correct the significance level for this many tests

5.c.3 If you want to change it, run through the previous steps again and check your conclusions.

# rerun all A/B test with Bonferroni correction
# test control group vs test group 
print('Bonferroni corrected control group 246 vs test group 248\n')
test_hypothesis(exp_a = '246', exp_b = '248', alpha = bonferonni_correction)

# test control group vs test group 
print('Bonferroni corrected control group 247 vs test group 248\n')
test_hypothesis(exp_a = '247', exp_b = '248', alpha = bonferonni_correction)

# test combined control group vs test group 
print('Bonferroni corrected control groups combined 249 (246 + 247) vs test group 248\n')
test_hypothesis(exp_a = '249', exp_b = '248', alpha = bonferonni_correction)

Bonferroni corrected control group 246 vs test group 248

Event: CartScreenAppear:
	p-value:  0.07842923237520116
	Reject the null hypothesis for CartScreenAppear, there is a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		50.97%	48.48% 

Event: MainScreenAppear:
	p-value:  0.2949721933554552
	Reject the null hypothesis for MainScreenAppear, there is a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		98.63%	98.27% 

Event: OffersScreenAppear:
	p-value:  0.20836205402738917
	Reject the null hypothesis for OffersScreenAppear, there is a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		62.08%	60.35% 

Event: PaymentScreenSuccessful:
	p-value:  0.2122553275697796
	Reject the null hypothesis for PaymentScreenSuccessful, there is a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		48.31%	46.55% 

Event: Tutorial:
	p-value:  0.8264294010087645
	Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		11.19%	11.00% 

Bonferroni corrected control group 247 vs test group 248

Event: CartScreenAppear:
	p-value:  0.5786197879539783
	Fail to reject the null hypothesis for CartScreenAppear, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		49.26%	48.48% 

Event: MainScreenAppear:
	p-value:  0.4587053616621515
	Reject the null hypothesis for MainScreenAppear, there is a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		98.53%	98.27% 

Event: OffersScreenAppear:
	p-value:  0.9197817830592261
	Fail to reject the null hypothesis for OffersScreenAppear, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		60.49%	60.35% 

Event: PaymentScreenSuccessful:
	p-value:  0.7373415053803964
	Fail to reject the null hypothesis for PaymentScreenSuccessful, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		46.08%	46.55% 

Event: Tutorial:
	p-value:  0.765323922474501
	Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		11.26%	11.00% 

Bonferroni corrected control groups combined 249 (246 + 247) vs test group 248

Event: CartScreenAppear:
	p-value:  0.18175875284404386
	Reject the null hypothesis for CartScreenAppear, there is a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		50.11%	48.48% 

Event: MainScreenAppear:
	p-value:  0.29424526837179577
	Reject the null hypothesis for MainScreenAppear, there is a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		98.58%	98.27% 

Event: OffersScreenAppear:
	p-value:  0.43425549655188256
	Reject the null hypothesis for OffersScreenAppear, there is a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		61.28%	60.35% 

Event: PaymentScreenSuccessful:
	p-value:  0.6004294282308704
	Fail to reject the null hypothesis for PaymentScreenSuccessful, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		47.19%	46.55% 

Event: Tutorial:
	p-value:  0.764862472531507
	Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		11.23%	11.00% 

5.c.3 Conclusion

• 8 out of 15 tests reject the null hypothesis and are considered signficantly different after performing tests with the Bonferonni Correction
  • 4 out of 5 in 246 vs 248 test (MainScreenAppear, OfferScreenAppear, CartScreenAppear, PaymentScreenSuccessful)
  • 1 out of 5 in 247 vs 248 test (MainScreenAppear)
  • 3 out of 5 in 249 (246+247) vs 248 test (MainScreenAppear, OfferScreenAppear, CartScreenAppear)
• this is roughly half of the tests, suggesting that with out the correction our data has several false positives
  • the test with control group 246 vs test group 248 has the most false positives
    • over all, they had a over a 2% difference in their data (see section 5.a), so this may be the reason for this. The difference between control 247 and test 248 was less than 1% (see section 5.b)
    • because the the combo control group 249 vs test group 248 includes 246, this may explain why it too has several false positives

Reviewer's comment Bomferonni is an excellent choice!

Step 5. Conclusion

• 4 groups of tests were performed:

  • A/A
    • Control groups 246 and 247 have a 1.17% difference in users
      • this is a little bit more than the standard 1% for a successful A/A test and suggests that the data is not quite collected evenly
  • A1/B
    • Control group 246 and test group 248 have a 2% difference in users
      • this is more than the 1% standard and suggests that the data was not collected evenly
  • A2/B
    • Control group 247 and test group 248 have a 0.96% difference in users
      • this is close to 1% difference suggesting it was collected relatively evenly
  • A1+A2/B

• A function was created to calculate the number and shares of users per group per event and tested if they were significantly different

  • when performing these tests with out the Bonferonni Correction, all tests were determined to be not significantly different, suggesting that the change in font type does not affect user behavior
• The Bonferonni Correction was calculated for 15 tests
  • this value is 0.536708769840247
  • and represents a 53.67% probability of at least one false positive result out of 15 tests
    • this is very high, but as you increase the number of tests, so do you increase the number of false positives
  • when the Bonferonni Correction was applied to the above test combinations 8 out of 15 tests where revealed to be false positives
    • this suggests that the fonts do have an affect on users behavior
      • however it is mostly between control group 246 vs test group 248, which has a 2% difference in users, so the data is not collected evenly
      • compare this to control group 247 vs test group 248 which has less than a 1% difference in users, only one of five tests had a false positive, suggesting that the font does not affect user behavior
      • Because this data (247 vs 248) is most similar, I would say this data is more accurate compared to data with control group 246 and should be used to make a decision - which is that the font change does not affect user behavior

extra test to compare the control groups with the bonferonni correction

# Bonferonni correction test: control group vs control group 
print('Bonferroni corrected control group 246 vs control group 247\n')
test_hypothesis(exp_a = '246', exp_b = '247', alpha = bonferonni_correction)

Bonferroni corrected control group 246 vs control group 247

Event: CartScreenAppear:
	p-value:  0.22883372237997213
	Reject the null hypothesis for CartScreenAppear, there is a signficant difference between the proportions of experiments 246 and 247.
	Experiment:		246	247
	Number of users:	2484	2513
	Share of users:		50.97%	49.26% 

Event: MainScreenAppear:
	p-value:  0.7570597232046099
	Fail to reject the null hypothesis for MainScreenAppear, there is not a signficant difference between the proportions of experiments 246 and 247.
	Experiment:		246	247
	Number of users:	2484	2513
	Share of users:		98.63%	98.53% 

Event: OffersScreenAppear:
	p-value:  0.2480954578522181
	Reject the null hypothesis for OffersScreenAppear, there is a signficant difference between the proportions of experiments 246 and 247.
	Experiment:		246	247
	Number of users:	2484	2513
	Share of users:		62.08%	60.49% 

Event: PaymentScreenSuccessful:
	p-value:  0.11456679313141849
	Reject the null hypothesis for PaymentScreenSuccessful, there is a signficant difference between the proportions of experiments 246 and 247.
	Experiment:		246	247
	Number of users:	2484	2513
	Share of users:		48.31%	46.08% 

Event: Tutorial:
	p-value:  0.9376996189257114
	Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments 246 and 247.
	Experiment:		246	247
	Number of users:	2484	2513
	Share of users:		11.19%	11.26% 

6 Overall Conclusion

• User behavior on company app was examined in this report
• Sales/event funnel was evaluated
• A/A/B test on font styles were performed

General Information

• The data evaluates 3 groups: 2 controls, 1 test
  • the data covers 7 days: 2019-08-01 to 2019-08-07
  • the data accounts for 7551 unique users
    • the number of users per group are not quite evenly split (1% or less difference is preferred)
      • controls 246 and 247 have a 1.17 % difference
      • control 246 vs test 248 have a 2.1% difference
      • control 247 vs test 248 have a 0.96% difference

Sales/Event Funnel

• The data reveals a general sales/event funnel:
  • 1. MainScreenAppear
  • 1. OfferScreenAppear
  • 1. CartScreenAppear
  • 1. PaymentScreenSuccessful
  • Tutorial is optional
• 47% of users make it to PaymentScreenSuccessful
  • this is a little bit less than half of users, so there is room for improvement
  • however nearly 95% of users who add something to their cart end up making a purchase
• 38% of users leave before making it to the OfferScreenAppear page

A/A/B test on font styles

• The following tests were performed:
  • A/A
  • A1/B
  • A2/B
  • A1+A2/B

• Initially these test were performed with out the Bonferonni Correction.

  • this resulted in 15 tests (excluding the A/A tests)
  • and revealed that there are no statisitical differences between the groups for any of the actions in the funnel
    • suggesting the font change does not affect user behavior

• The Bonferonni Correction was calculated:

  • 0.536708769840247
    • which suggests a 53.67% probability of at least one false positive result out of 15 tests
  • after perfroming the tests with the Bonferonni Correction:
    • 8 out 15 were revealed to be false positives, i.e. the null hypothesis was rejected and there was a significant difference between the groups
    • due to this one may think that the font actually does impact user behavior, however 7 of these false positives involved data between control group 246 vs test group 248 which has a 2% difference in amount of users:
      • 4 out of 5: 246 vs 248 (A1/B)
      • 3 out of 5: 246 +247 vs 248 (A1+A2/B)
    • the test comparing control group 247 vs test group 248 has only 1 false positive.
      • while this false positive does occur on the MainScreenAppear, which is likely the first page users see, it is likely not affecting user behavior since there is no difference in the amount of purchses being made between these groups

General Conclusion

• Ultimately I do not think the fonts are affecting user behavior
• However I think that one of the control groups was not collected properly for comparison against the test group
  • this is supported when the two control groups are tested with the Bonferonni Correction, this leads to 3 out of 5 false positives including the PaymentScreenSuccessful action

Recommendations

• one of two things can happen going forward
  • 1. Accept the results of control group 247 vs test 248 suggesting that fonts do not change user behavior
  • 1. Redo the tests with A/A test data more accurately collected

Report conclusion update post completion of the program

• The reviewer from Project 13.1 (Final Project: A/B Test Project) explained that calculated the Bonferonni Correction incorrectly, which was similar to how I ran my calculations in this project.

Updates to the summary above

• The correct Bonferonni Correction value is:
  • 0.0033333333333333335
  • calculated by the desired_alpha / num_tests
• The correct FWER (not the Bonferonni Correction) is 0.04885001789563237
  • which suggests about a 5% probability of at least one false positive result out of 15 tests
• With the correct significance leve the results of ALL the tests fail to reject the null hypothesis, there is not a significant difference between groups. The font change does not affect user behavior

# performed post program completion
# correct bonferonni correction after learning the right way from project 12.1
correct_bon_corr = .05 / 15 # desired alpha / number of tests
print('correct Bonferonni Correction: {}'.format(correct_bon_corr))

correct Bonferonni Correction: 0.0033333333333333335

# performed post program completion
# calculate the FWER
correct_fwer = 1 - (1 - correct_bon_corr) ** num_tests
print('FWER for these experiments is {}, this alpha to be used in the Bonferroni correction.'.format(correct_fwer))
print('This means there is a {:.2%} probability of at least one false positive result out of {} tests'.format(correct_fwer, num_tests))

FWER for these experiments is 0.04885001789563237, this alpha to be used in the Bonferroni correction.
This means there is a 4.89% probability of at least one false positive result out of 15 tests

# performed post program completion
# rerun all A/B test with CORRECT Bonferroni correction
# test control group vs test group 
print('Corrected: Bonferroni corrected control group 246 vs test group 248\n')
test_hypothesis(exp_a = '246', exp_b = '248', alpha = correct_bon_corr)

# test control group vs test group 
print('Corrected: Bonferroni corrected control group 247 vs test group 248\n')
test_hypothesis(exp_a = '247', exp_b = '248', alpha = correct_bon_corr)

# test combined control group vs test group 
print('Corrected: Bonferroni corrected control groups combined 249 (246 + 247) vs test group 248\n')
test_hypothesis(exp_a = '249', exp_b = '248', alpha = correct_bon_corr)

Corrected: Bonferroni corrected control group 246 vs test group 248

Event: CartScreenAppear:
	p-value:  0.07842923237520116
	Fail to reject the null hypothesis for CartScreenAppear, there is not a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		50.97%	48.48% 

Event: MainScreenAppear:
	p-value:  0.2949721933554552
	Fail to reject the null hypothesis for MainScreenAppear, there is not a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		98.63%	98.27% 

Event: OffersScreenAppear:
	p-value:  0.20836205402738917
	Fail to reject the null hypothesis for OffersScreenAppear, there is not a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		62.08%	60.35% 

Event: PaymentScreenSuccessful:
	p-value:  0.2122553275697796
	Fail to reject the null hypothesis for PaymentScreenSuccessful, there is not a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		48.31%	46.55% 

Event: Tutorial:
	p-value:  0.8264294010087645
	Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments 246 and 248.
	Experiment:		246	248
	Number of users:	2484	2537
	Share of users:		11.19%	11.00% 

Corrected: Bonferroni corrected control group 247 vs test group 248

Event: CartScreenAppear:
	p-value:  0.5786197879539783
	Fail to reject the null hypothesis for CartScreenAppear, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		49.26%	48.48% 

Event: MainScreenAppear:
	p-value:  0.4587053616621515
	Fail to reject the null hypothesis for MainScreenAppear, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		98.53%	98.27% 

Event: OffersScreenAppear:
	p-value:  0.9197817830592261
	Fail to reject the null hypothesis for OffersScreenAppear, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		60.49%	60.35% 

Event: PaymentScreenSuccessful:
	p-value:  0.7373415053803964
	Fail to reject the null hypothesis for PaymentScreenSuccessful, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		46.08%	46.55% 

Event: Tutorial:
	p-value:  0.765323922474501
	Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments 247 and 248.
	Experiment:		247	248
	Number of users:	2513	2537
	Share of users:		11.26%	11.00% 

Corrected: Bonferroni corrected control groups combined 249 (246 + 247) vs test group 248

Event: CartScreenAppear:
	p-value:  0.18175875284404386
	Fail to reject the null hypothesis for CartScreenAppear, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		50.11%	48.48% 

Event: MainScreenAppear:
	p-value:  0.29424526837179577
	Fail to reject the null hypothesis for MainScreenAppear, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		98.58%	98.27% 

Event: OffersScreenAppear:
	p-value:  0.43425549655188256
	Fail to reject the null hypothesis for OffersScreenAppear, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		61.28%	60.35% 

Event: PaymentScreenSuccessful:
	p-value:  0.6004294282308704
	Fail to reject the null hypothesis for PaymentScreenSuccessful, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		47.19%	46.55% 

Event: Tutorial:
	p-value:  0.764862472531507
	Fail to reject the null hypothesis for Tutorial, there is not a signficant difference between the proportions of experiments 249 and 248.
	Experiment:		249	248
	Number of users:	4997	2537
	Share of users:		11.23%	11.00% 

Reviewer's comment So, you are right. New fonts are not important. We made a lot of tests and choose good alpha and confirm it by code. Thank you for recommendationd and for sum up all here. Perfect conclusion! :)